feat(ltx-2.5): LTX-2.5 joint video+audio DiT, and a video seam that is no longer MiniMax-only (#435) - #641

…urce, and the USAGE anchor rotted in this branch's own rebase (#608, #649)

Fresh-review findings on #643. Records only: no `src/`, `include/` or `tests/`
file is touched and no row state moves.

F1 - the USAGE.md anchor rotted during this branch's own rebase. The spec cited
`docs/USAGE.md:902` for the parser-name count. That was right at the pre-rebase
base `a20cdac30`; #641 (`cefacd2d0`) changed `docs/USAGE.md` by 572 lines and
the count line moved to `:1126`. Re-derived and READ at this head:

    1126: | `--tool-call-parser <name>` | `hermes` | Tool-call dialect
          (41 names over 37 families). `auto` detects from the chat template ...

`:902` at this head is a container-tag table row (`| :main-cuda / -vulkan /
-cpu | moves with main ...`). The spec now says `:1126` and carries the reason
it moved, because nothing gates anchor line numbers (#632): a rebase moves them
silently and a re-run preflight still passes, which is exactly how this
survived. The PR body's "every anchor was re-derived and read at this head" is
corrected there; it was not true of this one.

F2 - the new characterisation was wrong where the old one was right. W0's COUNT
correction stands: five upstream-only names, not the "three Rust/Harmony-backed
ones" `docs/STATUS.md` claimed. Its restatement that
`openai`/`cohere_command3`/`cohere_command4` are "plain Python" does not, and it
is wrong in the more dangerous direction - plain Python is not the same as
portable. Read in the pinned checkout 5559679229bc961848b121ccdeaa8fa5d79bec98:

  * vllm/tool_parsers/gptoss_tool_parser.py:17 - `GptOssToolParser` is a
    declared stub ("All output parsing is handled by HarmonyParser. This stub
    exists as a capability declaration via HarmonyParser.tool_parser_cls"), and
    BOTH methods, :31 and :45, raise NotImplementedError("GptOssToolParser is a
    stub. Use HarmonyParser for tool parsing."). Harmony-backed by explicit
    declaration, not a text parser.
  * vllm/tool_parsers/cohere_command_tool_parser.py:6-13 - both Cohere classes
    (base :34, :125, :138) are shims over the out-of-tree `cohere_melody`
    package (`PyFilter`/`PyFilterOptions`), with a hard ImportError if it is
    absent. All parsing is delegated; NO grammar exists in vLLM source to port.
  * minimax_m3_tool_parser.py:7,18 - `RustToolParser` +
    rust_parser_name = "MinimaxM3ToolParser". The grammar is in the Rust crate.
  * inkling_tool_parser.py:4,7 - a ParserEngine adapter over
    `InklingParserToolAdapter`.

So of the five upstream-only names, only `inkling` is portable from vLLM
source. That is load-bearing, so W1/W2 are RESCOPED, not reworded. W1 is three
different jobs: `inkling` is the ordinary port; `minimax_m3` is written from the
wire format its docstring describes, recorded from-scratch in
porting-inventory.md section 9, with its upstream test as the fidelity gate; and
`openai` has no upstream text parser at all, so it owes a decision recorded in
the spec before code - mirror the Harmony seam, or register the name with an
explicit refusal naming the missing piece, per AGENTS.md. W2's Cohere scope
carries an undisclosed external dependency and likewise owes a decision rather
than a port. `docs/STATUS.md` and the `roadmap_v1.md` #608 row move to the same
reading, since AGENTS.md requires the three surfaces to agree.

F3 - filed as #649 rather than repaired here. The `TOOLS-CALLING-CORE` row
(.agents/engine-matrix.md:156) still records `tool_parser_names()` 40 /
`reasoning_parser_names()` 7; re-derived at this head they are 41
(src/vllm/entrypoints/openai/tool_parsers/abstract.cpp:269) and 12
(src/vllm/entrypoints/openai/reasoning_parsers/abstract.cpp:72). It was flagged
in the PR body with no issue behind it, which is the thing AGENTS.md forbids.
#649 now tracks it and is linked from the roadmap_v1.md open-issues table. It is
not fixed in this commit: it is a third row's prose and its reasoning half
belongs to SAMPLE-REASONING (#605), so repairing it inside a records-only
TOOLS-PARSER-BREADTH change is the "would surprise a reviewer" case the in-flow
rule carves out. Code and tests are already correct - test_detect.cpp:221 pins
41 - so only the record drifted.

F4 was informational. The PR body cited `TOOLS-STRUCTURAL-TAG` as the `PARTIAL`
precedent, but that row carries `planned: specs/structural-tag.md` and no such
file exists, so it is PARTIAL WITHOUT a committed spec - the opposite of the
reason argued. The body is corrected; the `PARTIAL` state on this row is
unchanged and remains correct on its own argument.

scripts/agent-preflight.sh --staged: exit 0, "All gates green".
scripts/check-windows-portability.py: exit 1, INHERITED - three errors in
src/vllm/multimodal/video_engine.cpp introduced by #641, tracked by #648 (the
fix) and #646 (the checker runs in no CI job, hook or preflight).
`git diff origin/main -- src/ include/ tests/ scripts/` is empty and
video_engine.cpp is byte-identical to origin/main.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]

…L, and the issue table still held the retracted premise (#607)

Fresh-review findings on #642. Records only: one spec section and two
roadmap_v1.md rows. No src/, include/ or tests/ change, no row-state change.

F1 - the seam map omitted the ENFORCEMENT consumer. Section 1.5 mapped the
memory consequence (interfaces.py:293) and the kernel gate (qwen3_next.py:325)
but not what makes a limit a limit. Added to the seam table, each line read in
the pinned checkout 5559679229bc961848b121ccdeaa8fa5d79bec98:

  * vllm/multimodal/processing/context.py:409-428 -
    MultiModalProcessingInfo.validate_num_items raises
    VLLMValidationError("At most {limit} {modality}(s) may be provided in one
    prompt."), appending " Set `--limit-mm-per-prompt` to increase this limit."
    at :425-426 when the MODEL supports more than the user allowed.
  * its two call sites: context.py:461 inside parse_mm_data (:430), and
    vllm/entrypoints/chat_utils.py:662, whose only escape is enable_mm_embeds
    plus a *_embeds modality at limit 0 (chat_utils.py:653-660).
  * context.py:392-405 - allowed_mm_limits folds the USER limit with the
    model's supported_mm_limits by min(), so a user limit never raises a
    model's ceiling.
  * further consumers: vllm/multimodal/registry.py:126 and
    vllm/v1/worker/encoder_cudagraph.py:139.

Following that chain changes what the flag DOES: get_limit_per_prompt returns 0
for every modality (multimodal.py:321-327), so validate_num_items computes
limit = 0 and raises for any request carrying one or more items. Upstream
--language-model-only REFUSES every multimodal request. The refusal is now
placed explicitly in L1, with the reason it belongs there rather than in L3: it
is the limits' own semantics, and a limit nothing enforces is not a limit.

That retires my own overly generous claim. The section said "L2 without L3 is
honest and shippable: the flag would be accepted and would correctly zero the
limits". The honesty caveat I attached guarded only "frees VRAM", leaving the
larger divergence unguarded - we would ACCEPT an image request upstream
REFUSES, which is the flag's main observable behaviour. The claim is now
conditional: with L1's refusal L2 ships honestly and only the memory win is
owed; without it L2 is not shippable at all, because a flag that is accepted and
inert is worse than the abort it replaces - an abort is visible and a silently
served image request is not.

F2 - one of the three mandated surfaces still stated the retracted premise.
AGENTS.md requires the issue, the row's spec and the PR body to agree. The
issue title was corrected; .agents/roadmap_v1.md's #607 row was not. It showed
the Row column as "-" and still described the flag as "43 recipes skip the
vision encoder to hand its VRAM to the KV cache". Now placed on
ENG-MM-INPUT-PIPELINE and re-described from the corrected mechanism, naming
BOTH consequences. Verified there is exactly ONE #607 row in that table
(grep -c '^| \[#607\]' == 1), because two branches adding the same keyed row
merge cleanly and define it twice. The recipe-surface sweep table's own
--language-model-only line carried the same "free encoder VRAM" framing and is
reconciled with it.

F3 - wrong emphasis in a citation. The seam row labelled "--limit-mm-per-prompt
/ --language-model-only" cited arg_utils.py:555,1276,1691, all three of which
are the language_model_only lines. Since section 1.5's thesis is that the limits
come first, the two flags now have a row each and both triples are cited, all
six read at the pin:

    555  language_model_only: bool = MultiModalConfig.language_model_only
    556  limit_mm_per_prompt: dict[str, int | dict[str, int]] = get_field(
    1276     "--language-model-only", **multimodal_kwargs["language_model_only"]
    1279     "--limit-mm-per-prompt", **multimodal_kwargs["limit_per_prompt"]
    1691     language_model_only=self.language_model_only,
    1692     limit_mm_per_prompt=self.limit_mm_per_prompt,

scripts/agent-preflight.sh --staged: exit 0, "All gates green".
scripts/check-windows-portability.py: exit 0, "Windows portability contract OK"
- this branch is based on a20cdac, BEFORE #641 added
src/vllm/multimodal/video_engine.cpp, so it does not carry that inherited red
(#648 / #646). It will appear on a rebase onto current main, and it is not this
branch's: no src/ file is touched here.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]

FOLLOWING_AGENTS_PROTOCOL

main advanced to b81a5e4 (#641's LTX-2.5, then #650's oracle policy) and
roadmap_v1.md's Open-issues table conflicted. That table is a KEYED RECORD, so
AGENTS.md forbids resolving it by automatic three-way merge: took main's file
WHOLESALE and reapplied this branch's two scoped edits.

The two edits are NOT the same operation, and treating them alike is the trap I
walked into first. #608 already exists on main -- it landed via #612 -- so it is an
UPDATE IN PLACE. #649 is new, so it appends. My first attempt appended both and
produced a duplicate #608, caught by counting rather than by any gate: the
uniqueness check the checker does run would have caught it too, but only after I
had already written the wrong thing.

Verified rather than asserted: diff against main is +2/-1 -- one row replaced, one
added -- and grep gives exactly one #608 row and one #649 row.

One gate needed a second look. test_cpu_x86_llamacpp_floor's contended-leg case
failed once here at load 26.97, then passed twice consecutively in this same
worktree, and passes on clean main at the same load; the test file and tools/bench
are byte-identical to main. That is #618's known flakiness, not this resolution.
Re-ran the full staged gate afterwards for a clean result rather than reasoning
past the red.

scripts/agent-preflight.sh --staged: exit 0, all gates green.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]

…, and rescope W1/W2 (#608, #649) (#643)

FOLLOWING_AGENTS_PROTOCOL

The row said "Qwen-Coder XML, Mistral, pythonic, and remaining parsers", "Our
code: -", INVENTORIED, with a spec that was never written. All three parsers it
named by title were implemented, alongside 38 others. This is a RECORD REPAIR, not
an achievement: nothing new ships here.

Three counts were wrong and two of them cancelled, which is why they survived a
spec, a merge and a report. Ours is 41, not 40 -- the audit's regex used
[a-z0-9_]+ and silently dropped the hyphenated granite-20b-fc, while
test_detect.cpp:221 has asserted 41 since MODEL-MUSE-GLIMMER-W7. Upstream is 44,
not 43. And two of OUR names exist in no upstream registry at the pin (qwen3, a
local alias, and muse_glimmer, the off-pin unmerged-PR exception recorded in
porting-inventory.md §16). 44 - 5 = 39 shared, + 2 = 41. The old "40 = 43 - 3"
balanced only because two errors cancelled.

The five upstream-only names are confirmed, but the characterisation of them was
wrong in the dangerous direction -- "plain Python" reads as portable. Reading the
method bodies rather than the imports: GptOssToolParser is a declared stub whose
methods raise NotImplementedError and defer to Harmony; both Cohere classes
hard-ImportError without the out-of-tree cohere_melody package and delegate all
parsing to it. So only `inkling` is portable from vLLM source. W1/W2 are RESCOPED
accordingly -- one ordinary port, one from-scratch write gated on the upstream
test, and two names that each owe a recorded decision before any code.

State INVENTORIED -> PARTIAL, not ANCHOR-BACKFILL: that state asserts a missing
leaf spike and the spec is committed, while PARTIAL's "known to omit upstream
behavior" is exactly true of the five. Summary counts re-derived from the rows.

Reviewed twice. The first review FAILed it on four record findings -- a
docs/USAGE.md anchor that rotted during this branch's own rebase (#641 moved that
file by 572 lines), the portability characterisation, a defect flagged only in a
PR body, and a backwards precedent citation. All four repaired by a fresh
implementer; #649 filed for the stale TOOLS-CALLING-CORE counts. The scoped
re-review returned PASS with no findings.

Records only: 4 markdown files, no src/, no tests/, no checker weakened.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]

FOLLOWING_AGENTS_PROTOCOL

This branch was based on a20cdac and main has since reached 67e53e7 -- #641's
LTX-2.5, #650's oracle policy, and #643's own tool-parser backfill. roadmap_v1.md
conflicted.

Resolved by the keyed-record rule rather than by three-way merge: took main's file
WHOLESALE, then reapplied this branch's two edits IN PLACE -- the #607 issue row
and the recipe-sweep table's --language-model-only line, both of which existed
already and both of which this branch UPDATES rather than adds. Diff against main
is +2/-2, exactly two replacements and no additions.

The reason wholesale mattered here and not just as ceremony: this branch's tree
still carried the PRE-CORRECTION #608 row, which said four of the five parsers
were portable at the pin. #643 has since corrected that to "only inkling is
portable from vLLM source". An automatic merge would have been free to keep either
side; taking main's file wholesale means the corrected row is the one that
survives, and I checked that it did rather than trusting the mechanism.

Exactly one #607 row and one #608 row afterwards.

scripts/agent-preflight.sh --staged: exit 0, all gates green.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]

FOLLOWING_AGENTS_PROTOCOL

Mechanical rebase by the operator so a PASSED review can land. The row is one of
three reds blocking every open PR in the repository, and the only thing standing
between it and main was a conflict in two SHARED RECORD files -- the exact
"shared file = lock" pattern AGENTS.md names. No code conflict.

Both keyed records resolved BY KEY: main's version taken wholesale, this row's
scoped edits reapplied, unrelated keys verified byte-identical to main.

The defect this row fixes was introduced by the LTX-2.5 lane (cefacd2, #641),
which this operator coordinates; the fix and its review are another session's
work, and the review is a fresh one that verified the seam is a real
alignment-safe memcpy rather than a silenced diagnostic.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5[1m] [claude-code]

# Conflicts:
#	.agents/roadmap_v1.md
#	docs/FEATURES.md

… uint16_t* it may not form

FOLLOWING_AGENTS_PROTOCOL

`main` has been RED on `sanitize-cpu (address,undefined)` since cefacd2 (#641, the LTX-2.5 landing). This closes it.

`Ltx2LoadVaeWeights` read the mapping through a pointer it is not allowed to form: `StTensor::data` points at `8 + <JSON header length> + <sum of preceding tensor sizes>` (safetensors_reader.h:16-23) and NOT ONE of those three terms is required to be even, so the `reinterpret_cast<const uint16_t*>` is UB regardless of whether the load happens to work on x86. Reads now go through `vt::LoadUnaligned<uint16_t>` -- a plain `std::memcpy` into a local, no packed attribute, no `-fno-sanitize=alignment`, no char* round-trip that re-forms the same access. The sanitizer configuration is untouched, so no gate was widened.

THIRD recurrence of one class: #301 (closed; it is what left the `vt::LoadUnaligned` seam behind) and #627 (still open) preceded it. The coverage that caught it was ACCIDENTAL -- the fixture's JSON header happens to land that tensor odd -- so the row owed a case that FORCES the odd offset and asserts the parity, and it has one.

Reviewed PASS by a fresh reviewer who answered the only question that mattered with a measurement rather than an argument: post-fix seam at an ALIGNED address over all 65536 bit patterns MISMATCHES=0 exit 0; post-fix at an ODD address returns 0x3322 matching memcpy, exit 0; pre-fix cast at an odd address aborts with "load of misaligned address", exit 1. It also mutated the test's own parity guard -- selecting the EVEN fixture while keeping the odd REQUIRE goes RED -- so the guard is armed rather than decorative, and confirmed values are unchanged on the aligned path.

Sibling sweep with a positive control: across the ltx2 + minimax_h3 families (34 files) no live cast survives; the same regex fires on olmo2_weights.cpp and qwen3_5_weights.cpp, proving the instrument works. 19 sites / 11 loaders remain elsewhere, inventoried on #627 -- this row closes one instance of the class and says so rather than implying the class is closed. A defect in that inventory was reported to #627 during the review: several sites dismissed as "over std::vector::data(), suitably aligned" are actually `OwnedBytes`, which may hold a BORROWED GGUF mmap view.

Operator rebase and gate: the only thing between a PASSED review and main was a conflict in two SHARED RECORD files, resolved BY KEY -- main's version wholesale, this row's edits reapplied, each file verified to differ from main by exactly its own one line. check-device-leakage OK, DSR 32 == baseline 32, and `git diff origin/main` on the checker and its baseline is EMPTY. BUILD_EXIT=0, 0 errors, 0 ENOSPC, REGISTERED=448, ctest 447/448, and `test_ltx2_video` -- the suite that was red under UBSan -- 31 cases / 513 assertions exit 0. The single ctest failure is #737, main's own Music3 golden red, which reproduces on a clean main.

Closes #674.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5[1m] [claude-code]